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Abstract Text ( 1) : 

A method for localization refinement of inspection patterns comprises the steps of 
providing a template image comprising pixels in a pattern, each pixel having an 
intensity, providing an input image having a same pattern of pixels as the template 
image and calculating an energy function by weighting a sum of modified optical 
flow constraints at locations of the pixels of both the template image and the 
input image to determine a shift and rotation between the pattern of the template 
image and the input image . 

Application Filing Date (1) : 
19971218 

Brief Summary Text (8) : 

In one proposal, an image registration technique is performed by fitting feature 
points in the zero-crossings extracted from the image to be inspected to the 
corresponding points extracted from the CAD model via an affine transformation. 
Unfortunately, the correspondence between the two set of feature usually cannot be 
reliably obtained. Another proposal employed a sum-of -squared -dif f erences (SSD) 
method to determine the shift between the two images. In addition to its 
restriction on the recovery of shift alignment only, this method could not handle 
illumination changes between the image to be inspected and the reference image. 

Brief Summary Text (12) : 

A method for localization refinement of inspection patterns includes the steps of 
providing a template image comprising pixels in a pattern, each pixel having an 
intensity, providing an input image having a same pattern of pixels as the template 
image and minimizing an energy function formed by weighting a sum of modified 
optical flow constraints at locations of the pixels of both the template image and 
the input image to determine a shift and rotation between the pattern of the 
template image and the input image. 

Brief Summary Text (13) : 

In other methods, the steps may include partitioning the template image into blocks 
of pixels, determining a reliability measure for each pixel in each block and 
identifying the pixel location for each block having a largest reliability measure 
as the feature point for each block. 

Brief Summary Text (17): 

In other methods, the step of determining feature points may include partitioning 
the template image into blocks of pixels, determining a reliability measure for 
each pixel in each block and identifying the pixel location for each block having a 
largest reliability measure as the feature point for each block. The step of 
minimizing an energy function formed by weighting a sum of modified optical flow 
constraints at locations of the feature points further includes the steps of 
calculating a Hessian matrix and a gradient vector of the energy function based on 
an initial guess of a shift and a rotation, updating the initial guess based on the 
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calculating the Hessian matrix and the gradient vector of the energy function and 
iteratively recalculating the Hessian matrix and the gradient vector of the energy 
function until an updated guess is within an acceptable increment from a previous 
updated guess. The step of smoothing the template image to reduce noise effects may 
also be included. The step of minimizing an energy function formed by weighting a 
sum of modified optical flow constraints at locations of the feature points may 
further include the step of incorporating an illumination change factor into the 
optical flow constraints for accounting for pixel intensity changes due to 
illumination effects. 

Brief Summary Text (18) : 

A method for computer inspection for determining misalignment between inspection 
patterns includes the steps of providing a template image comprising blocks of 
pixels, determining features points on the template image from among blocks of 
pixels by selecting a pixel location in each block having a largest reliability 
measure, averaging pixels in an area surrounding each feature point to reduce noise 
in the template image, providing an input image with an initial shift and rotation 
guess to the template image, minimizing an energy function formed by weighting a 
sum of modified optical flow constraints at locations of the feature points of the 
template image to determine a shift and rotation between the template image and the 
input image. The step of minimizing an energy function formed by weighting a sum of 
modified optical flow constraints at locations of the feature points may further 
include the steps of calculating a Hessian matrix and a gradient vector of the 
energy function based on an initial guess of a shift and a rotation, updating the 
initial guess based on the calculating the Hessian matrix and the gradient vector 
of the energy function and iteratively recalculating the Hessian matrix and the 
gradient vector of the energy function until an updated guess is within an 
acceptable increment from a previous updated guess. The step of minimizing an 
energy function formed by weighting a sum of modified optical flow constraints at 
locations of the feature points may further include the step of incorporating an 
illumination change factor into the optical flow constraints for accounting for 
pixel intensity changes due to illumination effects. 

Detailed Description Text (5) : 

A modified optical flow constraint is derived herein by using a generalized 
brightness assumption and replacing displacement vectors by the 2D rigid 
transformation parameters. Based on the consideration of efficiency as well as 
reliability, a feature point selection method is used to choose a set of locations 
with reliable modified optical flow constraints. The method is primarily based on 
an iterative energy minimization formulation with an energy function being a 
weighted sum of modified optical flow constraints at selected locations. The method 
handles large degrees of illumination changes by explicitly incorporating an 
illumination change factor into the modified optical flow constraint. The 
minimization of the energy function is accomplished via an efficient iterative 
algorithm, which has proved to be very reliable for small localization refinement 
from experiments described herein, namely for position errors within 5 pixels and 
rotation angles within 5 degrees. The minimization process is basically a very 
efficient search of the best transformation, i.e. shift and rotation, for general 
template matching. 

Detailed Description Text (33) : 

This gives a sum-of-squared -dif f erences error, which is a measure of absolute 
distance between points within each block. Since it is more reasonable to use a 
measure of minimum distance, the sum of squared linear approximation errors sse(x, 
y) is normalized by using the magnitude of the gradient of the linear 
function . alpha . x-K beta . y+. gamma . to obtain a measure of minimum distance, i.e. sse 
(x, y) /( .alpha. .sup. 2 + . beta . . sup . 2 ) . A reliability measure .sigma.(x, y) of using 
the modified optical flow constraint at location (x, y) may be defined as follows: 
##EQU12## 
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Detailed Description Text (42): 

Referring now in specific detail to the drawings in which like reference numerals 
identify similar or identical elements throughout the several views, and initially 
to FIG. 1, a flow chart 10 of localization refinement steps of the method in 
accordance with the disclosure of the present invention is illustrated. The 
implementation of a localization refinement algorithm includes two phases, namely a 
training phase 6 and execution phase 8. The training phase 6 includes a template 
object step 12, where a template object is recovered, a preprocessing or smoothing 
step 14, a feature selection step 16 which includes the computation of the 
gradients to determine the feature point locations for the template image. The 
computation involved in this phase is completely related to the template image 
only, therefore it can be performed in advance of the capture of the input image. 
Since the templates in many industrial inspection applications were already known 
prior to starting the localization task, the training phase 6 for these templates 
can be pre-computed beforehand. 

Detailed Description Text (44): 

The execution phase 8 is basically an iterative energy minimization process step 20 
to estimate the shift and rotation angle difference between an input image from 
step 22 and the template image. The shift and rotation angle difference are 
outputted in step 24. The total time required in the execution phase 8 of the 
algorithm is independent of the size of the image. It depends on the number of 
feature points and the number of iterations required in the energy minimization 
process. In our experiments, the execution phase takes about 20. about. 30 msec on a 
multi-user SUN SPARC-20 workstation, depending on the number of iterations required 
to achieve the convergence in the energy minimization process. 

Detailed Description Text (46) : 

Referring to FIG. 1C, step 20 of FIG. 1A may include substeps for calculating the 
Hessian matrix and the gradient vector of the energy function (E or E ! ). Details of 
these steps are described above. In step 34, modified optical constraints are 
calculated for each pixel or each feature point. In step 36, the modified optical 
flow constraints are weighted. The Hessian matrix and the gradient vector of E or 
E ! is calculated in step 38 based on an initial guess (. DELTA. x. sup .( 0 ) , 
. DELTA. y. sup. (0) , . theta . . sup . ( 0 ) ) from step 22. A new guess is calculated using 
the Newton method, for example, in step 40. In steps 42 and 44, the new guess is 
compared to the previous guess to determine if an acceptable increment has been 
achieved, for example, the changes in the x- and y- shifts are within about 0.1 
pixels and the change in the rotation angle is within about 0.1 degrees. If the 
guess is within the acceptable increment, (. DELTA. x, . DELTA. y, .theta.) are 
outputted in step 24 (FIG. 1A) . If the guess is not within the acceptable 
increment, steps 38, 40, 42 and 44 are repeated. 

Detailed Description Text (48) : 

Referring to FIGS. 2A, an illustrative inspection pattern is shown for a "pound 
sign". In FIGS. 2B and 2C, transformations were randomly generated (including 
shifts and rotations) for each template with the shifts in the x and y directions 
randomly selected from a uniform distribution ranging from -5 to 5 pixels and the 
rotation angles drawn from a uniform distribution between -5 to 5 degrees. Global 
illumination changes were simulated by randomly drawing samples for the 
coefficients a and b in the illumination change model from the uniform 
distributions of the ranges [0.75, 1.25] and [30, -30]. To simulate a noise effect, 
a random is multiplicative noise was imposed to each pixel with each multiplication 
factor randomly selected from two different uniform distributions in the ranges 
between -10% to 10% of multiplicative noise (FIG. 2B) , and -20% to 20% of 
multiplicative noise (FIG. 2C) , to show the accuracy of the estimation under 
different levels of noises. After the above simulation processes, the intensity 
values were rounded to the closest integer values between 0 and 255. 

Detailed Description Text (49) : 
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To alleviate the noise effect, a simple smoothing operation may be performed on the 
template image simply by averaging in a 3. times. 3 window, for example. Then, the 
iterative energy minimization method is applied to estimate the shift and rotation. 
In one example, with the convergence criterion that the changes in the x- and y- 
shifts are within 0.1 pixels and the change in the rotation angle is within 0.1 
degrees, the minimization usually converges in about 10 iterations. 

Detailed Description Paragraph Table (1) : 

TABLE 1 Mean Standard Deviation Errors in . DELTA. x -0.014 pixels 0.035 pixels 
Errors in . DELTA. y -0.015 pixels 0.047 pixels Errors in .theta. -0.006 degrees 
0.019 degrees 

Detailed Description Paragraph Table (2) : 

TABLE 2 Mean Standard Deviation Errors in . DELTA. x -0.016 pixels 0.057 pixels 
Errors in . DELTA. y -0.016 pixels 0.065 pixels Errors in .theta. -0.008 degrees 
0.028 degrees 

Detailed Description Paragraph Table (3) : 

TABLE 3 Mean Standard Deviation Errors in . DELTA. x -0.026 pixels 0.035 pixels 
Errors in . DELTA. y -0.003 pixels 0.016 pixels Errors in .theta. -0.002 degrees 
0.013 degrees 

Detailed Description Paragraph Table (4) : 

TABLE 4 Mean Standard Deviation Errors in . DELTA. x -0.027 pixels 0.042 pixels 
Errors in . DELTA. y -0.010 pixels 0.030 pixels Errors in .theta. -0.004 degrees 
0.015 degrees 



CLAIMS : 



1. A method for localization refinement of inspection patterns comprising the steps 
of: 



providing a template image comprising pixels in a pattern, each pixel having an 
intensity; partitioning the template image into blocks of pixels; 

determining a reliability measure for each pixel in each block, the reliability 
measure being based upon a sum of the squares error and a polynomial fitting 
function for the pixels in the block; 

identifying a pixel location for each block having a largest reliability measure as 
a feature point for each block; 

providing an input image having a same pattern of pixels as the template image, the 
input image having an initial shift and rotation relative to the template image; 

modifying optical flow constraint equations to account for illumination changes in 
the input image; and 

minimizing an energy function formed by weighting a sum of the modified optical 
flow constraints at locations of the pixels of both the template image and the 
input image to determine a shift and rotation between the pattern of the template 
image and the input image for determining misalignments between the template image 
and the input image for an inspection process the minimizing step being performed 
at locations of the feature points of the template image to determine the shift and 
rotation between the template image and the input image. 

5. A method for localization refinement of inspection patterns comprising the steps 
of: 



determining features points on a template image from among blocks of pixels by: 
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partitioning the template image into blocks of pixels; 

determining a reliability measure for each pixel in each block, the reliability 
measure being based upon a sum of the squares error and a polynomial fitting 
function for the pixels in the block; 

identifying the pixel location for each block having a largest reliability measure 
as the feature point for each block; 

providing an input image with an initial shift and rotation relative to the 
template image; 

modifying optical flow constraint equations to account for illumination changes in 
the input image; and 

minimizing an energy function formed by weighting a sum of the modified optical 
flow constraints at locations of the feature points of the template image to 
determine a shift and rotation between the template image and the input for 
determining misalignments between the template image and the input image for an 
inspection process, the minimizing step being performed at locations of the feature 
points of the template image to determine the shift and rotation between the 
template image and the input image. 

9. A method for computer inspection for determining misalignment between an 
inspection pattern an object with a pattern to be inspected comprising the steps 
of: 

providing a template image for the inspection pattern comprising blocks of pixels; 

determining features points on the template image from among blocks of pixels by 
selecting a pixel location in each block having a largest reliability measure the 
reliability measure being based upon a sum of the squares error and a polynomial 
fitting function for the pixels in the block; 

averaging pixels in an area surrounding each feature point to reduce noise in the 
template image; 

providing an input image for the object with a pattern to be inspected with an 
initial shift and rotation guess with respect to the template image; 

modifying optical flow constraint equations to account for illumination changes in 
the input image; and 

minimizing an energy function formed by weighting a sum of the modified optical 
flow constraints at locations of the feature points of the template image to 
determine a shift and rotation between the template image and the input image for 
determining misalignments between the template image and the input image for an 
inspection process. 
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